Tokenization platform

ABSTRACT

A tokenization platform and method is described for accurately tokenizing character strings, including but not limited to non-delimited character strings of the type commonly used in Internet domain names and computer filenames, to accurately identify words and phrases occurring therein. In one embodiment, a phased tokenization approach is used in which the final phase is a lexical analysis-based tokenization using a dictionary. The dictionary may be advantageously created and updated based upon one or more query logs associated with respective information retrieval systems, thereby ensuring that the dictionary accurately reflects currently-used terminology and captures alternative spellings and presentations of words and phrases submitted by users.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.12/178,775, filed Jul. 24, 2008, the entirety of which is incorporatedby reference herein.

FIELD OF THE INVENTION

The present invention generally relates to automated systems and methodsfor tokenizing a string of characters to identify words and phrasesoccurring therein.

BACKGROUND

As used herein, the term “delimiter” refers to one or more charactersthat are used to specify a boundary between separate, independent wordsoccurring in a string of characters. In some character strings, nodelimiters are used to specify a boundary between words occurring withinthe string. Such non-delimited character strings are very commonly usedin Internet domain names and computer filenames. An example of anInternet domain name that includes a non-delimited character string is“www.digitalcamerareview.com.” In this domain name, the non-delimitedcharacter string “digitalcamerareview” includes the separate,independent words “digital,” “camera” and “review.” An example of acomputer filename that includes a non-delimited character string is“catinthehat.gif.” In this filename, the non-delimited character string“catinthehat” includes the separate, independent words “cat,” “in,”“the,” and “hat.” Each word identified within a non-delimited characterstring may have independent meaning. Furthermore, identified words takentogether may have meaning, in which case they form a phrase.

A non-delimited character string that forms a part of an Internet domainname may include words or phrases that provide valuable clues aboutaccessible subject matter within the corresponding Internet domain. Ifsuch words and phrases could be accurately identified, they could beused to improve the performance of Internet search engines or othersystems that match keywords or other information submitted by a user todomains on the World Wide Web. Likewise, a non-delimited characterstring that forms a part of a computer filename may include words orphrases that provide valuable clues about the information contained inor represented by a file identified by the filename. If such words andphrases could be accurately identified, they could be used to improvethe performance of search engines, desktop search tools, or othersystems that match keywords or other information submitted by a user tocomputer files.

What is needed then is a system and method for tokenizing characterstrings, including but not limited to non-delimited character strings ofthe type commonly used in Internet domain names and computer filenames,to accurately identify words and phrases occurring therein.

BRIEF SUMMARY OF THE INVENTION

A tokenization platform and method is described herein for accuratelytokenizing character strings, including but not limited to non-delimitedcharacter strings of the type commonly used in Internet domain names andcomputer filenames, to accurately identify words and phrases occurringtherein.

In particular, a method for tokenizing a character string is describedherein. In accordance with the method: (a) it is determined whetherthere are any words or phrases in a dictionary that match a series ofcharacters within the character string that begins at the firstcharacter of the character string; (b) for each matching word or phraseidentified in step (a), the matching word or phrase is assigned to atokenization path, wherein the tokenization path comprises one or morecontiguous words or phrases embedded within the character string, and acorresponding series of characters is removed from the beginning of thecharacter string, thereby generating a shortened character stringassociated with the tokenization path or terminating the tokenizationpath; (c) if no matching word or phrase is identified in step (a), thenany tokenization path with which the character string is associated isterminated; (d) steps (a), (b) and (c) are recursively performed for anyshortened character string generated in step (b) until all tokenizationpaths are terminated; (e) for any tokenization path formed through theperformance of steps (a)-(d), a score is calculated based on each wordor phrase assigned to the tokenization path; and (f) the word(s) and/orphrase(s) associated with a tokenization path having the highest scoreare selected as tokens associated the character string.

A further method for tokenizing a character string is described herein.In accordance with the method, a dictionary is populated with words andphrases included in a set of search queries submitted by users of one ormore information retrieval systems over a first predetermined timeperiod. One or more series of characters within the character string arethen identified that match a word or phrase populated within thedictionary. The foregoing method may further include periodicallyupdating the dictionary with words and phrases included in additionalsets of search queries submitted by users of the one or more informationretrieval systems over predetermined time periods that are subsequent tothe first predetermined time period.

A computer program product is also described herein. The computerprogram product comprises a computer-readable medium having computerprogram logic recorded thereon for enabling a processing unit totokenize a character string. The computer program logic includes firstmeans, second means, third means, fourth means, fifth means and sixthmeans. The first means are for enabling the processing unit to determineif there are any words or phrases in a dictionary that match a series ofcharacters within the character string that begins at the firstcharacter of the character string. The second means are for enabling theprocessing unit to assign each matching word or phrase identified by thefirst means to a tokenization path, wherein the tokenization pathcomprises one or more contiguous words or phrases embedded within thecharacter string, and to remove a corresponding series of charactersfrom the beginning of the character string, thereby generating ashortened character string associated with the tokenization path orterminating the tokenization path. The third means are for enabling theprocessing unit to terminate any tokenization path with which thecharacter string is associated if no matching word or phrase isidentified by the first means. The fourth means are for enabling theprocessing unit to recursively perform the functions associated with thefirst means, the second means and the third means with respect to anyshortened character string generated by the second means until alltokenization paths are terminated. The fifth means are for enabling theprocessing unit to calculate a score for any tokenization path formed bythe execution of the first means, the second means, the third means andthe fourth means based on each word or phrase assigned to thetokenization path. The sixth means are for enabling the processing unitto select the word(s) and/or phrase(s) associated with a tokenizationpath having the highest score as tokens associated the character string.

A further computer program product is described herein. The computerprogram product comprises a computer-readable medium having computerprogram logic recorded thereon for enabling a processing unit totokenize a character string. The computer program logic includes firstmeans and second means. The first means are for enabling the processingunit to populate a dictionary with words and phrases included in a setof search queries submitted by users of one or more informationretrieval systems over a first predetermined time period. The secondmeans are for enabling the processing unit to identify one or moreseries of characters within the character string that match a word orphrase populated within the dictionary. The computer program logic mayfurther include means for enabling the processing unit to periodicallyupdate the dictionary with words and phrases included in additional setsof search queries submitted by users of the one or more informationretrieval systems over predetermined time periods that are subsequent tothe first predetermined time period.

Further features and advantages of the invention, as well as thestructure and operation of various embodiments of the invention, aredescribed in detail below with reference to the accompanying drawings.It is noted that the invention is not limited to the specificembodiments described herein. Such embodiments are presented herein forillustrative purposes only. Additional embodiments will be apparent topersons skilled in the relevant art(s) based on the teachings containedherein.

BRIEF DESCRIPTION OF THE DRAWINGS/FIGURES

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate the present invention and, togetherwith the description, further serve to explain the principles of theinvention and to enable a person skilled in the relevant art(s) to makeand use the invention.

FIG. 1 is a block diagram of a tokenization platform in accordance withan embodiment of the present invention.

FIG. 2 is a block diagram that illustrates various elements of thetokenization platform of FIG. 1 in accordance with one embodiment of thepresent invention.

FIG. 3 depicts a system in accordance with an embodiment of the presentinvention in which a tokenization platform is configured to populate adictionary with words and phrases derived from a plurality of querylogs.

FIG. 4 depicts an example logical structure of dictionary used by atokenization platform in accordance with one embodiment of the presentinvention.

FIG. 5 is a block diagram of a tokenization platform in accordance withan embodiment of the present invention in which a tokenization engineapplies a phased tokenization technique to identify one or more tokensbased on a character string.

FIG. 6 depicts a flowchart of a phased tokenization technique foridentifying one or more tokens based on a character string in accordancewith an embodiment of the present invention.

FIG. 7 depicts a flowchart for performing selective alphanumeric-basedtokenization in accordance with one embodiment of the present invention.

FIG. 8 depicts a flowchart of a method for performing lexicalanalysis-based tokenization using a dictionary in accordance with oneembodiment of the present invention.

FIG. 9 illustrates the logical structure of a simple prefix tree.

FIG. 10 illustrates the logical structure of a simple prefix tree inwhich nodes immediately below a root of the prefix tree store 4- and5-letter prefixes.

FIG. 11 is an illustration of multiple tokenization paths that may beformed through the application of a lexical analysis-based tokenizationprocess to the input character string “mariasaintsunglasshr” inaccordance with one embodiment of the present invention.

FIG. 12 is a block diagram of a computer system that may be used toimplement one or more aspects of the present invention.

The features and advantages of the present invention will become moreapparent from the detailed description set forth below when taken inconjunction with the drawings, in which like reference charactersidentify corresponding elements throughout. In the drawings, likereference numbers generally indicate identical, functionally similar,and/or structurally similar elements. The drawing in which an elementfirst appears is indicated by the leftmost digit(s) in the correspondingreference number.

DETAILED DESCRIPTION OF THE INVENTION A. Introduction

The following detailed description refers to the accompanying drawingsthat illustrate exemplary embodiments of the present invention. However,the scope of the present invention is not limited to these embodiments,but is instead defined by the appended claims. Thus, embodiments beyondthose shown in the accompanying drawings, such as modified versions ofthe illustrated embodiments, may nevertheless be encompassed by thepresent invention.

References in the specification to “one embodiment,” “an embodiment,”“an example embodiment,” or the like, indicate that the embodimentdescribed may include a particular feature, structure, orcharacteristic, but every embodiment may not necessarily include theparticular feature, structure, or characteristic. Moreover, such phrasesare not necessarily referring to the same embodiment. Furthermore, whena particular feature, structure, or characteristic is described inconnection with an embodiment, it is submitted that it is within theknowledge of one skilled in the art to implement such feature,structure, or characteristic in connection with other embodimentswhether or not explicitly described.

B. Example Tokenization Platform in Accordance with an Embodiment of thePresent Invention

FIG. 1 is a block diagram of a tokenization platform 100 in accordancewith an embodiment of the present invention. Generally speaking,tokenization platform 100 operates to receive a character string 102 andto identify one or more tokens 104 within the character string.Character string 102 may represent any discrete series of characters forwhich tokenization is to be performed. For example, character string 102may comprise an Internet domain name or a computer filename, althoughthese examples are not intended to be limiting. Token(s) 104 representsone or more separate characters sequences within character string 102that have been identified by tokenization platform 100. A token mayrepresent, for example, a character sequence within character string 102that has been matched to a word or phrase by tokenization platform 100.A token may also represent other types of character sequences as well.

By way of example, tokenization platform 100 may receive a characterstring 102 corresponding to the Internet domain name“www.tarzanlordofthejungle.com” and identify the corresponding tokens104: “www,” “tarzan,” “lord,” “of,” “the,” “jungle” and “com.” Asanother example, tokenization platform 100 may receive a characterstring 102 corresponding to the computer filename“mariasaintsunglasshr031.jpg” and identify the corresponding tokens 104:“maria,” “saint,” “sunglass,” “hr,” “031,” and “jpg.”

Depending upon the implementation, tokenization platform 100 may beimplemented in software, through the execution of program instructionsby one or more general purpose or special-purpose processors, inhardware using analog and/or digital circuits, or as a combination ofsoftware and hardware. An example of a computer system that may be usedto implement a software-based implementation of tokenization platform100 will be described below in reference to FIG. 12.

Each character in character string 102 may be encoded with a characterencoding scheme such as ASCII (American Standard Code for InformationInterchange) or UTF-8 (8-bit Universal Character Set/UnicodeTransformation Format) to facilitate automated processing of the stringby tokenization platform 100. Likewise, each character in the token(s)104 may be encoded using such a character encoding scheme.

FIG. 2 is a block diagram that illustrates various elements oftokenization platform 100 of FIG. 1 in accordance with one embodiment ofthe present invention. As shown in FIG. 2, tokenization platform 100includes a tokenization engine 202 and a dictionary 204 communicativelycoupled thereto. Dictionary 204 includes a structured collection ofwords and phrases. As used herein, the term “phrase” refers to a seriesof two or more words. Tokenization engine 202 is configured to receive acharacter string 102 and to identify one or more tokens 104 within thecharacter string. To perform this function, tokenization engine 202 isconfigured to compare character sequences within character string 102 towords and phrases stored in dictionary 204 in a manner that will bedescribed in more detail below.

FIG. 3 depicts a system 300 in which tokenization platform 100 isconfigured to populate dictionary 204 with words and phrases derivedfrom a plurality of query logs 310 ₁, 310 ₂, . . . , 310 _(n). Each ofquery logs 310 ₁, 310 ₂, . . . , 310 _(n) is intended to represent astored set of search queries submitted by users of a correspondinginformation retrieval system (not shown in FIG. 3) over somepredetermined time period. As will be appreciated by persons skilled inthe relevant art(s), a search query typically comprises one or morekeywords input by a user for the purpose of locating desired resourcesmade available to the user by an information retrieval system. In system300, tokenization platform 100 includes a dictionary generator 302 thatextracts words and phrases from the search queries stored in each ofquery logs 310 ₁, 310 ₂, . . . , 310 _(n) and stores such words andphrases in dictionary 204.

In one embodiment, dictionary generator 302 also calculates or otherwiseobtains a frequency for each word and phrase stored in dictionary 204.The frequency associated with a particular word or phrase may representthe total number of times the particular word or phrase appears within adistinct search query stored in query logs 310 ₁, 310 ₂, . . . , 310_(n) divided by the total number of distinct search queries storedwithin those logs. However, this is only an example and persons skilledin the relevant art(s) will appreciate that numerous other methods forobtaining a frequency associated with a word or phrase may be used. Inone embodiment, dictionary generator 302 will only store a phrase withindictionary 204 if the frequency associated with the phrase exceeds apredetermined threshold.

FIG. 4 depicts an example logical structure 400 of dictionary 204 inaccordance with one embodiment of the present invention. In accordancewith logical structure 400, dictionary 204 includes a plurality of words402 ₁, 402 ₂, . . . , 402 _(n) and corresponding word frequencies 412 ₁,412 ₂, . . . , 412 _(n) associated therewith. Furthermore, dictionary204 includes a plurality of phrases 404 ₁, 404 ₂, . . . , 404 _(m) andcorresponding phrase frequencies 414 ₁, 414 ₂, . . . , 414 _(m)associated therewith. Logical structure 400 is provided by way ofexample only and persons skilled in the relevant art(s) will appreciatethat a wide variety of logical structures may be used to store words andphrases, and frequencies associated therewith, in dictionary 204.

In one embodiment of the present invention, dictionary generator 302 isconfigured to periodically update dictionary 204 based on new sets ofsearch queries stored in query logs 310 ₁, 310 ₂, . . . , 310 _(n). Thenew sets of search queries may also be stored within different querylogs than query logs 310 ₁, 310 ₂, . . . , 310 _(n), depending upon theimplementation. This process may involve replacing all of the words andphrases stored in dictionary 204 with new words and phrases derived fromthe new sets of search queries. Alternatively, this process may involvedeleting a subset of the words and phrases stored in dictionary 204while also adding a new set of words and phrases to dictionary 204. Ineither case, dictionary generator 302 may calculate or otherwise obtainnew frequencies to be associated with the words and phrases stored indictionary 204 as part of the update process.

In accordance with such an embodiment, each update of dictionary 204introduces new words and phrases derived from search queries submittedby users over a predetermined time period, wherein the predeterminedtime period is more recent than (and subsequent to) a predetermined timeperiod associated with a previous set of search queries used to populatedictionary 204. This process advantageously ensures that the words andphrases stored in dictionary 204 accurately reflect more currently-usedterminology, including newly-coined words and phrases and words andphrases associated with current events. This process also advantageouslyensures that alternative spellings and presentations of words andphrases submitted by users may be captured within dictionary 204.Furthermore, because the process also associates a frequency of use withsuch words and phrases, it provides a constantly-updated measure ofwhich words and phrases are currently most popular with users. Suchdynamic dictionary generation based on recently-submitted user searchqueries provides for more accurate tokenization of character strings andpresents a distinct advantage as compared to the user of staticdictionaries that do not change over time.

Query logs 310 ₁, 310 ₂, . . . , 310 _(n) may be associated with avariety of different information retrieval systems and/or sub-systems.The generation of dictionary 204 based on query logs associated with avariety of different information retrieval systems and/or sub-systemsmay ensure that dictionary 204 includes a broad variety of words andphrases that derive from a variety of different information areas. Forexample, in one embodiment, query logs 310 ₁, 310 ₂, . . . , 310 _(n)include one or more of: a query log associated with an informationretrieval system configured to retrieve Web pages, a query logassociated with an information retrieval system configured to retrieveimages, and a query log associated with an information retrieval systemconfigured to retrieve news content. In a particular embodiment, querylogs 310 ₁, 310 ₂, . . . , 310 _(n) include a query log representativeof search queries submitted to a Web search engine (such as YAHOO!SEARCH) from the United States over a one-week period, a query logrepresentative of search queries submitted to an Internet image searchengine (such as YAHOO! IMAGE SEARCH) over a one-month period, a querylog representative of search queries submitted to an Internet newssearch engine (such as YAHOO! NEWS SEARCH) over a one-month period, anda query log representative of search queries submitted to a Web searchengine from the United Kingdom, Asia, Germany, France, Australia and NewZealand over a one month period. Such an implementation may be used togenerate a dictionary 204 that comprises approximately 1.5 millionkeywords. However, this is only one example and should not be used tolimit the present invention.

It is noted that dictionaries other than those derived from query logsmay also be used to implement the present invention, provided such otherdictionaries provide some sort of frequency associated with each word orphrase stored therein. In an embodiment in which the dictionary isderived from a corpus of documents (such as a corpus of indexed Webpages), a term frequency-inverse document frequency (TF-IDF) may beassociated with each word or phrase.

-   -   1. Phased Tokenization Technique in Accordance with an        Embodiment of the Present Invention

FIG. 5 is a block diagram of one implementation of tokenization platform100 in which tokenization engine 202 applies a phased tokenizationtechnique to identify token(s) 104 based on character string 102. Asshown in FIG. 5, tokenization engine 202 includes a series of logicblocks that are collectively configured to implement such a phasedtokenization approach. These logic blocks include a delimiter-basedtokenizer 502, a capitalization-based tokenizer 504, analphanumeric-based tokenizer 506, and a lexical analysis-based tokenizer508. Alphanumeric-based tokenizer 506 and lexical analysis-basedtokenizer 508 are each configured to access dictionary 204 to perform arespective tokenization function. The manner in which each of theselogic blocks operate to collectively implement a phased tokenizationapproach will now be described in reference to flowchart 600 of FIG. 6.

As shown in FIG. 6, the method of flowchart 600 begins at step 602 inwhich delimiter-based tokenizer 502 receives input character string 102and identifies one or more first tokens within character string 102based on any delimiters identified in the character string. As noted inthe Background Section, a delimiter refers to one or more charactersthat are used to specify a boundary between separate, independent wordsoccurring in a string of characters. Some commonly-used delimitersinclude periods (“.”), hyphens (“-”), forward slashes (“/”), backwardslashes (“\”), commas (“,”), underscores (“_”), exclamation marks (“!”),and the like. Thus, for example, delimiter-based tokenizer 502 mayreceive the character string “team-building.jpg” and identify the firsttokens “team,” “building,” and “jpg” based on the hyphen and periodpresent within the character string.

Any token(s) identified in this manner by delimiter-based tokenizer 502are passed to capitalization-based tokenizer 504. Such token(s) arerepresented in FIG. 5 as first token(s) 512. If delimiter-basedtokenizer 502 cannot detect any delimiters within character string 102,then it will pass the entire character string as a single first token512 to capitalization-based tokenizer 504.

At step 604, capitalization-based tokenizer 504 receives first token(s)512 and identifies one or more second tokens within each of firsttoken(s) 512 based on any capital letters identified in each of firsttoken(s) 512. Thus, for example, capitalization-based tokenizer 504 mayreceive the first token “LordOfTheRings” and identify the second tokens“lord,” “of,” “the” and “rings” based on the capital letters “L,” “O,”“T” and “R” present within the first token.

Any token(s) identified in this manner by capitalization-based tokenizer504 are passed to alphanumeric-based tokenizer 506. Such token(s) arerepresented in FIG. 5 as second token(s) 514. If capitalization-basedtokenizer 504 cannot detect any capital letters within a first token512, then it will pass the entire first token 512 as a single secondtoken 514 to alphanumeric-based tokenizer 506.

At step 606, alphanumeric-based tokenizer 506 receives second token(s)514 and selectively identifies one or more third tokens within each ofsecond token(s) 514 based on alphabetic and numeric charactercombinations present in each of second token(s) 514. In particular,alphanumeric-based tokenizer 506 examines each second token 514 todetermine if it comprises a character combination in which one or moreletters are followed by one or more numbers or vice versa. Suchcharacter combinations might include, for example, the strings“picture1,” “chapter3,” “U2,” or “49ers.”

For each second token 514 so identified, alphanumeric-based tokenizer506 performs a process that will now be described with reference toflowchart 700 of FIG. 7. As shown in FIG. 7, at step 702,alphanumeric-based tokenizer 506 attempts to match the alphabetic andnumeric character combination to a word or phrase in dictionary 204 and,at decision step 704, determines whether a match is found. If no matchis found, then alphanumeric-based tokenizer 506 will separate theidentified alphabetic and numeric character combination into analphabetic token and a numeric token as shown at step 708. Thus, forexample, if alphanumeric-based tokenizer 506 could not match the secondtoken “picture1” to an entry in dictionary 204, it would separate thesecond token into third tokens “picture” and “1.”

However, if alphanumeric-based tokenizer 506 does find a match for theidentified alphabetic and numeric character combination in dictionary204, then alphabetic-based tokenizer 506 will determine if the frequencyassociated with the matching word or phrase exceeds a predefinedthreshold as shown at decision step 706. If the frequency associatedwith the matching word or phrase does not exceed the predefinedthreshold, then alphanumeric-based tokenizer 506 will separate theidentified alphabetic and numeric character combination into analphabetic token and a numeric token as shown at step 708. However, ifthe frequency associated with the matching word or phrase does exceedthe predefined threshold, then alphanumeric-based tokenizer 506 willleave the identified alphabetic and numeric character combination intactas shown at step 710. Thus, for example, if alphanumeric-based tokenizer506 could match the second token “U2” to an entry in dictionary 204 andthe frequency associated with the entry “U2” in dictionary 204 exceededa predefined threshold, alphanumeric-based tokenizer 506 would leave thesecond token “U2” intact. This technique advantageously ensures thatvalid alphanumeric tokens are not separated by alphanumeric-basedtokenizer 506.

Any token(s) identified in the foregoing manner by alphanumeric-basedtokenizer 506 are passed to lexical analysis-based tokenizer 508. Suchtoken(s) are represented in FIG. 5 as third token(s) 514. Ifalphanumeric-based tokenizer 506 does not separate any alphabet andnumeric character combinations within a second token 514, then it willpass the entire second token 514 as a single third token 516 to lexicalanalysis-based tokenizer 508.

Returning now to the description of flowchart 600 of FIG. 6, at step608, lexical analysis-based tokenizer 508 receives third token(s) 514and performs a lexical analysis using dictionary 204 on each of thirdtoken(s) 514 to identify one or more final tokens within each of thethird token(s). One manner by which lexical analysis-based tokenizer 508operates to perform this function will be described in detail in thefollowing sub-section. Any token(s) identified in the manner to bedescribed are passed from lexical analysis-based tokenizer 508 astoken(s) 104.

-   -   2. Lexical Analysis-Based Tokenization in Accordance with an        Embodiment of the Present Invention

FIG. 8 depicts a flowchart 800 of a method for performing lexicalanalysis-based tokenization using a dictionary in accordance with oneembodiment of the present invention. The method of flowchart 800 may beimplemented by lexical analysis-based tokenizer 508 as part of a phasedtokenization approach described above in reference to FIGS. 5 and 6,although the invention is not so limited. Persons skilled in therelevant art(s) will readily appreciate that the method of flowchart 800may be performed in conjunction with tokenization functions other thanthose described above in reference to FIGS. 5 and 6 and may also beperformed independently of any other tokenization functions.

As shown in FIG. 8, the method of flowchart 800 begins at step 802, inwhich lexical analysis-based tokenizer 508 receives a character string.The character string may be, for example, a third token 516 passed tolexical analysis-based tokenizer 508 from alphanumeric-based tokenizer506, although the method is not so limited.

At step 804, lexical analysis-based tokenizer 508 determines if thereare any words or phrases within dictionary 204 that match a series ofcharacters within the character string that begins at the first of thecharacter string.

As shown at decision step 806, if it is determined that there are anymatching words or phrases, then control flows to step 808. At step 808,for each matching word or phrase, lexical analysis-based tokenizer 508assigns the matching word or phrase to a tokenization path and removes acorresponding series of characters from the beginning of the characterstring. The removal of the corresponding series of characters from thebeginning of the character string will either generate a shortenedcharacter string associated with the tokenization path if all theremaining letters in the original character string are not consumed bythe removal, or terminate the tokenization path if all the remainingletters in the original character string are consumed.

As further shown at decision step 806, if no matching words or phrasesare identified during step 804, then any tokenization path with whichthe character string is associated is terminated as shown at step 810.

After the conclusion of both step 808 and step 810, control flows todecision step 812 in which lexical analysis-based tokenizer 508determines whether all previously-formed tokenization paths have beenterminated. If lexical analysis-based tokenizer 508 determines that notall previously-formed tokenization paths have been terminated, thenlexical analysis-based tokenizer 508 recursively performs the logic inthe loop comprising steps 804, 806, 808, 810 and 812 for each shortenedcharacter string generated in step 806 until all previously-formedtokenization paths have been terminated as shown at step 814.

However, if lexical analysis-based tokenizer 508 determines at decisionstep 812 that all previously-formed tokenization paths have beenterminated then control flows to step 816. During step 816, lexicalanalysis-based tokenizer 508 calculates a score for anypreviously-formed and terminated tokenization path. The score for eachtokenization path is calculated based on each word/phrase assigned tothe path.

For example, in one embodiment, lexical analysis-based tokenizer 508calculates the score for a tokenization path by calculating a value foreach word or phrase assigned to the path in accordance with the formula:log(frequency*10)+(length+frequency/10)²wherein frequency represents a frequency associated with the word orphrase and length represents the length in characters of the word orphrase. Lexical analysis-based tokenizer 508 then sums the values socalculated for each word or phrase assigned to the tokenization path tocalculate the overall score for the path. The foregoing scoringtechnique takes into account both the frequency of words and phrasesappearing in a tokenization path as well as the length of such words orphrases appearing in the path. This is desirable since longer words andphrases typically will have a lower associated frequency. The foregoingscoring technique also has the effect of penalizing tokenization pathsthat have unmatched characters at the end of the string. The foregoingapproach to scoring tokenization paths is provided by way of exampleonly. Persons skilled in the relevant art(s) will appreciate that otherscoring functions may be used to practice the method.

At step 818, lexical analysis-based tokenizer 508 selects the word(s)and/or phrase(s) associated with the tokenization path having thehighest score as the tokens associated with the character stringoriginally received in step 802.

In one embodiment of the present invention, in order to facilitate thematching operation of step 804, each word and phrase in dictionary 204is stored in a prefix tree, which also may be referred to as a trie. Aswill be appreciated by persons skilled in the relevant art(s), a prefixtree is an ordered tree data structure can be used to store anassociative array in which the keys are characters strings. By way ofexample, FIG. 9 depicts a simple prefix tree 900 that stores the words“CAN,” “CAT,” “ROD” and “ROE.” Unlike a binary search tree, no node inprefix tree 900 stores the word associated with that node. Instead, theposition of a node within tree 900 shows which word it is associatedwith. All the descendants of one node have a common prefix of the stringassociated with that node, and the root of the tree (denoted root 902)is associated with the empty string. As will be appreciated by personsskilled in the art, a prefix tree that stores all the words and phrasesof dictionary 204 may include many more nodes than shown in prefix tree900.

In an embodiment in which all the words and phrases in dictionary 204are stored in a prefix tree, step 804 comprises traversing the prefixtree starting with the first character in the relevant character stringand proceeding from node to node based on the sequence of characters inthe character string until all matching words and phrases within theprefix tree are found. Using a prefix tree to perform the matchingprocess of step 804 increases the speed with which that step may beperformed.

In one embodiment, a special prefix tree structure is used in which thenodes that occur immediately below the root level of the prefix tree arepopulated with 4- and 5-lettered prefixes of words and phrases occurringin dictionary 204 in order to expedite the matching process. By way ofexample, FIG. 10 depicts a simple prefix tree 1000 in which the fourletter prefix “DECA” and the five letter prefix “SUPRE” are stored innodes that are at a level 1004 immediately below the root 1002 of prefixtree 1000. The 4-letter prefix “DECA” forms part of the words “DECADE”and “DECANT” stored in prefix tree 1000 and the 5-letter prefix “SUPRE”forms part of the word “SUPREME” stored in prefix tree 1000. As will beappreciated by persons skilled in the art, a prefix tree that stores allthe words and phrases of dictionary 204 would include many more nodesthan shown in prefix tree 1000.

FIG. 11 is an illustration 1100 of multiple tokenization paths (namely,tokenization paths 1102, 1104, 1106, 1108, 1110, 1112 and 1114) that maybe formed through the application of the tokenization process offlowchart 800 to the input character string “mariasaintsunglasshr” inaccordance with one embodiment of the present invention. The manner inwhich the tokenization paths shown in FIG. 11 were formed will now bedescribed with further reference to the tokenization process offlowchart 800, so that that process may be better understood.

At step 802, lexical analysis-based tokenizer 508 receives the characterstring “mariasaintsunglasshr.”

At step 804, lexical analysis-based tokenizer 508 determines if thereare any words or phrases within dictionary 204 that match a series ofcharacters within “mariasaintsunglasshr” beginning with the firstcharacter “m” of that character string. In one embodiment, this step mayinvolve traversing a prefix tree that stores the words and phrases ofdictionary 204, wherein a first level of nodes below a root of theprefix tree include 4- or 5-letter prefixes. In such an embodiment,words and phrases within dictionary 204 that start with the prefix“mari” and “maria” would be targeted for the search.

As shown in FIG. 11, during step 804, lexical analysis-based tokenizer508 finds the matching words “maria” and “mariasaint” within dictionary204. Thus, at decision step 806, control flows to step 808, during whichlexical analysis-based tokenizer 508 assigns each of these words to atokenization path and removes a corresponding series of characters fromthe beginning of the character string “mariasaintsunglasshr.” Thisresults in the generation of the shortened character string“saintsunglasshr” in association with the tokenization path [“maria”]and the generation of the shortened character string “sunglasshr” inassociation with the tokenization path [“mariasaint”].

At decision step 812, it is determined that the tokenization paths[“maria”] and [“mariasaint”] have not been terminated, so control flowsto step 814, in which the loop comprising steps 804, 806, 808, 810 and812 is recursively performed for each shortened character stringgenerated in step 806 (namely “saintsunglasshr” and “sunglass hr”) untilall previously-formed tokenization paths have been terminated. For thesake of brevity, each iteration of this loop will not be described;however, the generation of certain tokenization paths will be describedto facilitate understanding of the process.

For example, with respect to the tokenization path [“maria”], theapplication of the foregoing loop results in the matching of the words“saint” and “saints” from dictionary 204 to sequences of characters atthe beginning of the shortened character string “saintsunglasshr” andthe assignment of these matching words to the respective tokenizationpaths [“maria”, “saint”] and [“maria”, “saints”]. For the tokenizationpath [“maria”, “saint”], the shortened character string “sunglasshr” isnow subject to further loop iterations of the loop, resulting in theultimate formation of tokenization paths 1102, 1104 and 1106. However,for the tokenization path [“maria”, “saints”], the shortened characterstring “unglasshr” cannot be matched to any words or phrases indictionary 204, resulting in termination of that tokenization path,which is denoted tokenization path 1108. Termination of a path isindicated by a circle in FIG. 11. Depending upon the scoring techniqueused, the termination of tokenization path 1108 with left-over letters“unglasshr” can result in a penalty being applied to the scoresubsequently calculated for path 1108 during step 816. With theexception of tokenization path 1108, all of the other tokenization pathsshown in FIG. 11 end with the assignment of the final word “hr”, therebyconsuming all the letters in the original character string andterminating those paths.

Once all tokenization paths have been terminated, they are scored duringstep 816. As discussed above, the score for each tokenization path maybe calculated based on a frequency associated with each word or phraseassigned to the path and a length associated with each word or phraseassigned to the path. At step 818, the word(s) and/or or phrase(s) inthe tokenization path receiving the highest score are selected as thetokens associated with the string “mariasaintsunglasshr.” In FIG. 11, itis assumed that highlighted tokenization path 1106 has received thehighest score. Therefore, the selected tokens will be “maria,” “saint,”“sunglass” and “hr.”

C. Example Applications

The various systems and methods described above for performingtokenization of a character string may advantageously be used to supportor improve a variety of applications. For example, by accuratelytokenizing Internet domain names, an embodiment of the present inventioncan identify words and phrases that can be used by an informationretrieval system, such as an Internet search engine, to matchuser-submitted queries to domains on the World Wide Web. Likewise, byaccurately tokenizing computer filenames, an embodiment of the presentinvention can provide words and phrases that can be used by aninformation retrieval system to match user-submitted queries to files(such as, for example, image files) available on the World Wide Web.

Yet another example of an application that may utilize an embodiment ofthe present invention is a desktop search tool. In particular, byaccurately tokenizing computer filenames, an embodiment of the presentinvention can identify words and phrases that can be used by a desktopsearch tool to match user-submitted keywords to files stored on orotherwise accessible to a computer.

Another example of an application that may utilize an embodiment of thepresent invention is a domain match service that automatically generatesWeb pages for a user when the user types an invalid domain name intohis/her Web browser. Such a domain match service may attempt to includeadvertisements, links or other content of interest to the user withinthe automatically-generated Web page based on the invalid domain nameinput by the user. By accurately tokenizing the invalid domain name toidentify words and phrases therein, an embodiment of the presentinvention can provide the domain match service with keywords that can beused to help identify relevant content for inclusion within theautomatically-generated Web page.

The foregoing applications have been identified by way of example only.Persons skilled in the relevant art(s) will appreciate that many otherapplications may benefit from the features of the embodiments of thepresent invention described herein.

D. Example Computer System Implementation

The elements of tokenization platform 100 as depicted in FIGS. 1, 2, 3and 5 and the methods of flowcharts 600, 700 and 800 as respectivelydepicted in FIGS. 6, 7 and 8 may each be implemented by aprocessor-based computer system. An example of such a computer system1200 is depicted in FIG. 12.

As shown in FIG. 12, computer system 1200 includes a processing unit1204 that includes one or more processors. Processor unit 1204 isconnected to a communication infrastructure 1202, which may comprise,for example, a bus or a network.

Computer system 1200 also includes a main memory 1206, preferably randomaccess memory (RAM), and may also include a secondary memory 1220.Secondary memory 1220 may include, for example, a hard disk drive 1222,a removable storage drive 1224, and/or a memory stick. Removable storagedrive 1224 may comprise a floppy disk drive, a magnetic tape drive, anoptical disk drive, a flash memory, or the like. Removable storage drive1224 reads from and/or writes to a removable storage unit 1228 in awell-known manner. Removable storage unit 1128 may comprise a floppydisk, magnetic tape, optical disk, or the like, which is read by andwritten to by removable storage drive 1224. As will be appreciated bypersons skilled in the relevant art(s), removable storage unit 1228includes a computer usable storage medium having stored therein computersoftware and/or data.

In alternative implementations, secondary memory 1220 may include othersimilar means for allowing computer programs or other instructions to beloaded into computer system 1200. Such means may include, for example, aremovable storage unit 1230 and an interface 1226. Examples of suchmeans may include a program cartridge and cartridge interface (such asthat found in video game devices), a removable memory chip (such as anEPROM, or PROM) and associated socket, and other removable storage units1230 and interfaces 1226 which allow software and data to be transferredfrom the removable storage unit 1230 to computer system 1200.

Computer system 1200 may also include a communication interface 1240.Communication interface 1240 allows software and data to be transferredbetween computer system 1200 and external devices. Examples ofcommunication interface 1240 may include a modem, a network interface(such as an Ethernet card), a communications port, a PCMCIA slot andcard, or the like. Software and data transferred via communicationinterface 1240 are in the form of signals which may be electronic,electromagnetic, optical, or other signals capable of being received bycommunication interface 1240. These signals are provided tocommunication interface 1240 via a communication path 1242.Communications path 1242 carries signals and may be implemented usingwire or cable, fiber optics, a phone line, a cellular phone link, an RFlink and other communications channels.

As used herein, the terms “computer program medium” and “computerreadable medium” are used to generally refer to media such as removablestorage unit 1228, removable storage unit 1230 and a hard disk installedin hard disk drive 1222. Computer program medium and computer readablemedium can also refer to memories, such as main memory 1106 andsecondary memory 1220, which can be semiconductor devices (e.g., DRAMs,etc.). These computer program products are means for providing softwareto computer system 1200.

Computer programs (also called computer control logic, programminglogic, or logic) are stored in main memory 1206 and/or secondary memory1220. Computer programs may also be received via communication interface1240. Such computer programs, when executed, enable the computer system1200 to implement features of the present invention as discussed herein.Accordingly, such computer programs represent controllers of thecomputer system 1200. Where the invention is implemented using software,the software may be stored in a computer program product and loaded intocomputer system 1200 using removable storage drive 1224, interface 1226,or communication interface 1240.

The invention is also directed to computer program products comprisingsoftware stored on any computer readable medium. Such software, whenexecuted in one or more data processing devices, causes a dataprocessing device(s) to operate as described herein. Embodiments of thepresent invention employ any computer readable medium, known now or inthe future. Examples of computer readable mediums include, but are notlimited to, primary storage devices (e.g., any type of random accessmemory) and secondary storage devices (e.g., hard drives, floppy disks,CD ROMS, zip disks, tapes, magnetic storage devices, optical storagedevices, MEMs, nanotechnology-based storage device, etc.).

E. Conclusion

While various embodiments of the present invention have been describedabove, it should be understood that they have been presented by way ofexample only, and not limitation. It will be understood by those skilledin the relevant art(s) that various changes in form and details may bemade therein without departing from the spirit and scope of theinvention as defined in the appended claims. Accordingly, the breadthand scope of the present invention should not be limited by any of theabove-described exemplary embodiments, but should be defined only inaccordance with the following claims and their equivalents.

What is claimed is:
 1. A method implemented on a machine having at leastone processor, storage, and a communication platform connected to anetwork for tokenizing a non-delimited character string, comprising:creating a dictionary with words and phrases included in a set of searchqueries submitted via the network by users of one or more informationretrieval systems including an Internet search engine over a firstpredetermined time period; identifying one or more series of characterswithin the non-delimited character string that match a word or phrase inthe dictionary, wherein the identifying comprises: determining if thereis any word or phrase in the dictionary that matches a series ofcharacters that begins at a first character of the non-delimitedcharacter string, assigning, for each matching word or phrase, thematching word or phrase to a tokenization path comprising one or morecontiguous words or phrases embedded within the non-delimited characterstring, removing a corresponding series of characters from beginning ofthe non-delimited character string to generate a shortened characterstring or terminate the tokenization path, terminating any tokenizationpath associated with the non-delimited character string if no matchingword or phrase is identified by the determining, recursively performingthe above determining, assigning, removing, and terminating with respectto any shortened character string generated by the assigning andremoving until all tokenization paths are terminated to generate one ormore tokenization paths, and determining a selected tokenization pathfrom the one or more tokenization paths; and designating the word orphrase assigned to the selected tokenization path as a token associatedwith the non-delimited character string.
 2. The method of claim 1,further comprising: periodically updating the dictionary with words andphrases included in additional sets of search queries submitted by usersof the one or more information retrieval systems over predetermined timeperiods that are subsequent to the first predetermined time period. 3.The method of claim 1, further comprising: storing the words and phrasesin the dictionary into a prefix tree.
 4. The method of claim 3, whereinidentifying the one or more series of characters comprises: traversingthe prefix tree starting at the first character of the non-delimitedcharacter string and proceeding from node to node of the prefix treebased on the sequence of characters in the non-delimited characterstring until all matching words and phrases within the prefix tree arefound.
 5. The method of claim 3, wherein storing the words and phrasesin the dictionary into a prefix tree comprises: storing four letterprefixes and five letter prefixes associated with the words and phrasesin the dictionary as nodes immediately below a root of the prefix tree.6. The method of claim 1, wherein the one or more information retrievalsystems include one or more of: an information retrieval systemconfigured to retrieve web pages; an information retrieval systemconfigured to retrieve images; and an information retrieval systemconfigured to retrieve news content.
 7. A machine-readable tangible andnon-transitory medium having information for enabling a processing unitto tokenize a non-delimited character string, wherein the information,when read by the machine, causes the machine to enable the processingunit to perform the following: creating a dictionary with words andphrases included in a set of search queries submitted via a network byusers of one or more information retrieval systems including an Internetsearch engine over a first predetermined time period; identifying one ormore series of characters within the non-delimited character string thatmatch a word or phrase in the dictionary, wherein the identifyingcomprises: determining if there is any word or phrase in the dictionarythat matches a series of characters that begins at a first character ofthe non-delimited character string, assigning, for each matching word orphrase, the matching word or phrase to a tokenization path comprisingone or more contiguous words or phrases embedded within thenon-delimited character string, removing a corresponding series ofcharacters from beginning of the non-delimited character string togenerate a shortened character string or terminate the tokenizationpath, terminating any tokenization path associated with thenon-delimited character string if no matching word or phrase isidentified by the determining, recursively performing the abovedetermining, assigning, removing, and terminating with respect to anyshortened character string generated by the assigning and removing untilall tokenization paths are terminated to generate one or moretokenization paths, and determining a selected tokenization path fromthe one or more tokenization paths; and designating the word or phraseassigned to the selected tokenization path as a token associated withthe non-delimited character string.
 8. The medium of claim 7, whereinthe information, when read by the machine, further causes the machine toperform the following: enabling the processing unit to periodicallyupdate the dictionary with words and phrases included in additional setsof search queries submitted by users of the one or more informationretrieval systems over predetermined time periods that are subsequent tothe first predetermined time period.
 9. The medium of claim 7, whereinthe information, when read by the machine, further causes the machine toperform the following: enabling the processing unit to store the wordsand phrases in the dictionary into a prefix tree.
 10. The medium ofclaim 9, wherein the step of enabling the processing unit to identifyone or more series of characters comprises: enabling the processing unitto traverse the prefix tree starting at the first character of thenon-delimited character string and proceeding from node to node of theprefix tree based on the sequence of characters in the non-delimitedcharacter string until all matching words and phrases within the prefixtree are found.
 11. The medium of claim 9, wherein the step of enablingthe processing unit to store the words and phrases in the dictionaryinto a prefix tree comprises: enabling the processing unit to store fourletter prefixes and five letter prefixes associated with the words andphrases in the dictionary as nodes immediately below a root of theprefix tree.
 12. The medium of claim 7, wherein the one or moreinformation retrieval systems include one or more of: an informationretrieval system configured to retrieve web pages; an informationretrieval system configured to retrieve images; and an informationretrieval system configured to retrieve news content.
 13. A system,comprising: a processing unit; and a memory containing a program, which,when executed by the processing unit, performs a method for tokenizing anon-delimited character string, the method comprising: creating adictionary with words and phrases included in a set of search queriessubmitted via a network by users of one or more information retrievalsystems including an Internet search engine over a first predeterminedtime period; identifying one or more series of characters within thenon-delimited character string that match a word or phrase in thedictionary, wherein the identifying comprises: determining if there isany word or phrase in the dictionary that matches a series of charactersthat begins at a first character of the non-delimited character string,assigning, for each matching word or phrase, the matching word or phraseto a tokenization path comprising one or more contiguous words orphrases embedded within the non-delimited character string, removing acorresponding series of characters from beginning of the non-delimitedcharacter string to generate a shortened character string or terminatethe tokenization path, terminating any tokenization path associated withthe non-delimited character string if no matching word or phrase isidentified by the determining, recursively performing the abovedetermining, assigning, removing, and terminating with respect to anyshortened character string generated by the assigning and removing untilall tokenization paths are terminated to generate one or moretokenization paths, and determining a selected tokenization path fromthe one or more tokenization paths; and designating the word or phraseassigned to the selected tokenization path as a token associated withthe non-delimited character string.
 14. The system of claim 13, themethod further comprising: periodically updating the dictionary withwords and phrases included in additional sets of search queriessubmitted by users of the one or more information retrieval systems overpredetermined time periods that are subsequent to the firstpredetermined time period.
 15. The system of claim 13, furthercomprising: storing the words and phrases in the dictionary into aprefix tree.
 16. The system of claim 15, wherein storing the words andphrases in the dictionary into a prefix tree comprises: storing fourletter prefixes and five letter prefixes associated with the words andphrases in the dictionary as nodes immediately below a root of theprefix tree.
 17. The system of claim 13, wherein the one or moreinformation retrieval systems include one or more of: an informationretrieval system configured to retrieve web pages; an informationretrieval system configured to retrieve images; and an informationretrieval system configured to retrieve news content.
 18. The method ofclaim 1, wherein each of the words and phrases in the dictionary has anassociated frequency that is above a predetermined threshold.
 19. Themethod of claim 18, wherein the associated frequency for a particularword or phrase represents a ratio between the total number of times theparticular word or phrase appears within a distinct search query and thetotal number of distinct search queries submitted by the users.
 20. Themethod of claim 1, further including replacing at least a subset of thewords and phrases stored in the dictionary with words and phrasesincluded in an additional set of search queries submitted by the users.